Obtaining force information in a minimally invasive surgical procedure

ABSTRACT

Methods of and a system for providing force information for a robotic surgical system. The method includes storing first kinematic position information and first actual position information for a first position of an end effector; moving the end effector via the robotic surgical system from the first position to a second position; storing second kinematic position information and second actual position information for the second position; and providing force information regarding force applied to the end effector at the second position utilizing the first actual position information, the second actual position information, the first kinematic position information, and the second kinematic position information. Visual force feedback is also provided via superimposing an estimated position of an end effector without force over an image of the actual position of the end effector. Similarly, tissue elasticity visual displays may be shown.

CROSS REFERENCE TO RELATED APPLICATIONS

This non-provisional application claims the benefit under 35 U.S.C.§119(e) of provisional U.S. Pat. App. No. 61/204,083 (filed Dec. 31,2008), which is incorporated herein by reference.

This application is related to non-provisional U.S. patent applicationSer. No. 12/428,142 [entitled “Visual Force Feedback in a MinimallyInvasive Surgical Procedure”] (concurrently filed) and to provisionalU.S. Pat. App. No. 61/204,085 (filed Dec. 31, 2008), both of which areincorporated herein by reference.

BACKGROUND

Minimally invasive surgical (MIS) procedures have become more commonusing robotic (e.g., telerobotic) surgical systems. One example of sucha system is the Minimally invasive robotic surgery system described incommonly owned U.S. Pat. No. 7,155,315 (filed Dec. 12, 2005), entitled“Camera Referenced Control in a Minimally Invasive Surgical Apparatus.”

A common form of minimally invasive surgery is endoscopy. Endoscopicsurgical instruments in minimally invasive medical techniques generallyinclude an endoscope for viewing the surgical field and working toolsdefining end effectors. The working tools are similar to those used inconventional (open) surgery, except that the working end or end effectorof each tool is separated from its handle by an approximately 12-inchlong extension tube. Typical surgical end effectors include clamps,graspers, scissors, staplers, or needle holders, as examples.

To manipulate end effectors, a human operator, typically a surgeon,manipulates or otherwise commands a locally provided master manipulator.Commands to the master manipulator are translated as appropriate andsent to a slave manipulator that could be remotely deployed. The slavemanipulator then moves the end effector according to the user'scommands.

In order to duplicate the “feel” of actual surgery, force feedback maybe included in minimally invasive robotic surgical systems. To providesuch feedback, conventional systems have the remote slave manipulatorfeed back force information to the master manipulator, and that forceinformation is utilized to provide haptic feedback to the surgeon sothat the surgeon feels as if he or she is manipulating the end effectorsdirectly by hand.

SUMMARY

The following presents a simplified summary of some embodiments of theinvention in order to provide a basic understanding of the invention.This summary is not an extensive overview of the invention. It is notintended to identify key/critical elements of the invention or todelineate the scope of the invention. Its sole purpose is to presentsome embodiments of the invention in a simplified form as a prelude tothe more detailed description that is presented later.

In accordance with an embodiment, a method of providing forceinformation for a robotic surgical system is provided. The methodincludes storing first kinematic position information and first actualposition information for a first position of an end effector; moving theend effector via the robotic surgical system from the first position toa second position; storing second kinematic position information andsecond actual position information for the second position; andproviding force information regarding force applied to the end effectorat the second position utilizing the first actual position information,the second actual position information, the first kinematic positioninformation, and the second kinematic position information.

In accordance with another embodiment, a method of providing forceinformation for a robotic surgical system is provided. The methodincludes storing first kinematic position information from joint statesof a linkage while the linkage supports an end effector at a firstposition; storing first visual position information from a first imageof the end effector captured while the end effector is at the firstposition; and providing force information regarding force applied to theend effector at the first position utilizing a first offset between thefirst kinematic position information and the first visual positioninformation.

In accordance with yet another embodiment, an assembly for providingforce information within a robotic system is provided. The assemblyincludes a first input for first kinematic position information fromjoint states of a linkage while the linkage supports an end effector ata first position; a second input for actual position information; and amodule coupled with the first input and the second input, the moduleproviding force information regarding force applied to the end effectorat the first position in response to the first kinematic positioninformation and the first visual position information.

In accordance with still another embodiment, a system of providing avisual representation of force information in a robotic system isprovided. The system includes an image input for receiving an image ofan end effector while a force is applied to the end effector; a displaycoupled to the image input so as to present an actual position of theend effector under the applied force; and a processor coupled to thedisplay, the processor generating a second image representing aprojected position of the end effector offset from the first position soas to visually indicate the force, the processor transmitting the secondimage to the display so that the second image is superimposed with thefirst image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a top view of an operating room which includes a minimallyinvasive telesurgical system in accordance with an embodiment;

FIG. 2 is front view of a patient cart for the minimally invasivetelesurgical system of FIG. 1;

FIG. 3 is a block diagram representing components of the minimallyinvasive telesurgical system of FIG. 1;

FIG. 4 is a block diagram representing components for a computer for usein the minimally invasive telesurgical system of FIG. 1 in accordancewith an embodiment;

FIG. 5 is a flowchart representing steps for calculating force on an endeffector in accordance with an embodiment;

FIG. 6 is a diagrammatic representation of movement of an end effectorbetween positions A and B with force F resisting the movement;

FIG. 7 is a flowchart representing steps for displaying force inaccordance with an embodiment;

FIG. 8 is a side perspective view of an end effector and syntheticrepresentation of an end effector showing force in accordance with anembodiment;

FIG. 9 is a flowchart representing steps for displaying a syntheticmodel in accordance with an embodiment;

FIG. 10 is a side perspective view of a master controller in accordancewith an embodiment;

FIG. 11 is a flow chart representing steps for filtering forces to limitforce measurements primarily to tip forces in accordance with anembodiment; and

FIG. 12 is a flow chart representing steps for displaying deformation ofa tissue using variations in color in accordance with an embodiment.

DETAILED DESCRIPTION

In the following description, various embodiments of the presentinvention will be described. For purposes of explanation, specificconfigurations and details are set forth in order to provide a thoroughunderstanding of the embodiments. However, it will also be apparent toone skilled in the art that the present invention may be practicedwithout the specific details. Furthermore, well-known features may beomitted or simplified in order not to obscure the embodiment beingdescribed.

Referring now to the drawings, in which like reference numeralsrepresent like parts throughout several views, FIG. 1 shows a minimallyinvasive telesurgical system 20 having an operator station or surgeonconsole 30 in accordance with an embodiment. The surgeon console 30includes a viewer 32 where an image of a surgical site is displayed to asurgeon S. As is known, a support (not shown) is provided on which thesurgeon S can rest his or her forearms while gripping two mastercontrols 700 (FIG. 10), one in each hand. More controls may be providedif more end effectors are available, but typically a surgeon manipulatesonly two controls at a time and, if multiple end effectors are used, thesurgeon releases one end effector with a master control 700 and graspsanother with same master control. When using the surgeon console 30, thesurgeon S typically sits in a chair in front of the surgeon console,positions his or her eyes in front of the viewer 32, and grips themaster controls 700, one in each hand, while resting his or her forearmson the support.

A patient side cart 40 of the telesurgical system 20 is positionedadjacent to a patient P. In use, the patient side cart 40 is positionedclose to the patient P requiring surgery. The patient side cart 40typically is stationary during a surgical procedure, and it includeswheels or castors to render it mobile. The surgeon console 30 istypically positioned remote from the patient side cart 40 and may beseparated from the patient side cart by a great distance, even milesaway, but will typically be used within the same operating room as thepatient cart.

The patient side cart 40, shown in more detail in FIG. 2, typicallyincludes two or more robotic arm assemblies. Each arm assembly typicallyincludes unpowered, lockable “set up” joints and powered “manipulator”joints. In the embodiment shown in FIG. 2, the patient side cart 40includes four robotic arm assemblies 42, 44, 46, 48, but more or fewermay be provided. Each robotic arm assembly 42, 44, 46, 48 is normallyoperatively connected to one of the master controls of the surgeonconsole 30. Thus, movement of the robotic arm assemblies 42, 44, 46, 48is controlled by manipulation of the master controls.

One of the robotic arm assemblies, indicated by the reference numeral42, is arranged to hold an image capturing device 50, e.g., anendoscope, or the like. The endoscope or image capturing device 50includes a viewing end 56 at a remote end of an elongate shaft 54. Theelongate shaft 54 permits the viewing end 56 to be inserted through asurgery entry port of the patient P. The image capturing device 50 isoperatively connected to the viewer 32 of the surgeon console 30 todisplay an image captured at its viewing end 56.

Each of the other robotic arm assemblies 44, 46, 48 includes a surgicalinstrument or tool 60, 62, 64, respectively. The tools 60, 62, 64 of therobotic arm assemblies 44, 46, 48 include end effectors 66, 68, 70,respectively. The end effectors 66, 68, 70 are mounted on wrist memberswhich are pivotally mounted on distal ends of elongate shafts of thetools, as is known in the art. The tools 60, 62, 64 have elongate shaftsto permit the end effectors 66, 68, 70 to be inserted through surgicalentry ports of the patient P. Movement of the end effectors 66, 68, 70relative to the ends of the shafts of the tools 60, 62, 64 is alsocontrolled by the master controls of the surgeon console 30. Theinstruments are typically removably attached to the manipulator arms,and a mechanical interface between the instrument and the arm transmitsactuating forces to the instrument.

The telesurgical system 20 includes a vision cart 80. In an embodiment,the vision cart 80 includes most of the “core” computer equipment orother controls for operating the telesurgical system 20. As an example,signals sent by the master controllers of the surgeon console 30 may besent to the vision cart 80, which in turn may interpret the signals andgenerate commands for the end effectors 66, 68, 70 and/or robotic armassemblies 44, 46, 48. In addition, video sent from the image capturingdevice 50 to the viewer 34 may be processed by, or simply transferredby, the vision cart 80. In other embodiments, “core” computer equipmentfor the telesurgical system may be distributed in the surgeon consoleand patient side cart.

FIG. 3 is a diagrammatic representation of the telesurgical system 20.As can be seen, the system includes the surgeon console 30, the patientside cart 40, and the vision cart 80. In addition, in accordance with anembodiment, an additional computer 82 and display 84 are provided. Thesecomponents may be incorporated in one or more of the surgeon console 30,the patient side cart 40, and/or the vision cart 80. For example, thefeatures of the computer 82 may be incorporated into the vision cart 80.In addition, the features of the display 84 may be incorporated into thesurgeon console 30, for example, in the viewer 32, or maybe provided bya completely separate display or the surgeon console or on anotherlocation. In addition, in accordance with an embodiment, the computer 82may generate information that may be utilized without a display, such asthe display 84.

Although described as a “computer,” the computer 82 may be a componentof a computer system or any other software or hardware that is capableof performing the functions herein. Moreover, as described above,functions and features of the computer 82 may be distributed overseveral devices or software components. Thus, the computer 82 shown inthe drawings is for the convenience of discussion, and it may bereplaced by a controller, or its functions may be provided by one ormore components.

FIG. 4 shows components of the computer 82 in accordance with anembodiment. In the embodiment shown in the drawing, the computer 82includes a tool tracking component 90, a kinematic component 92, and aforce component 94. Briefly described, the tool tracking component 90and kinematic component 92 provide information to the force component94. The force component 94 combines or otherwise utilizes thisinformation and outputs a force output 96.

A positional component is included in or is otherwise associated withthe computer 82. The positional component provides information about aposition of an end effector, such as one of the end effectors 66, 68,70. In the embodiment shown in the drawings, the tool tracking component90 is the positional component, and it provides information about aposition of an end effector, such as the end effectors 66, 68, 70. By“position,” we mean at least one of the location and/or the orientationof the end effector. A variety of different technologies may be used toprovide information about a position of an end effector, and suchtechnologies may or may not be considered tool tracking devices. In asimple embodiment, the positional component utilizes video feed from theimage capturing device 50 to provide information about the position ofan end effector, but other information may be used instead of, or inaddition to, this visual information, including sensor information,kinematic information, any combination of these, or additionalinformation that may provide the position and/or orientation of the endeffectors 66, 68, 70. Examples of systems that may be used for the tooltracking component 90 are disclosed in U.S. Pat. App. Pub. No. US2006/0258938, entitled, “Methods and System for Performing 3-D ToolTracking by Fusion of Sensor and/or Camera Derived Data During MinimallyInvasive Robotic Surgery”; U.S. Pat. No. 5,950,629 (filed Apr. 28,1994), entitled “System for Assisting a Surgeon During Surgery”; U.S.Pat. No. 6,468,265 (filed Nov. 9, 1999), entitled “Performing CardiacSurgery Without Cardioplegia”; and U.S. Pat. App. Pub. No. US2008/0004603 A1 (filed Jun. 29, 2006), entitled “Tool Position andIdentification Indicator Displayed in a Boundary Area of a ComputerDisplay Screen.” In accordance with an embodiment, the tool trackingcomponent 90 utilizes the systems and methods described in commonlyowned U.S. Pat. App. No. 61/203,975 (filed Dec. 31, 2008), which isincorporated herein by reference. In general, the positional componentmaintains information about the actual position and orientation of endeffectors. This information is updated depending upon when theinformation is available, and may be, for example, asynchronousinformation.

To manipulate the tools 60, 62, 64, each of the slave manipulators inthe robotic arm assemblies 42, 44, 46, 48 is conventionally formed oflinkages that are coupled together and manipulated through motorcontrolled joints. These slave manipulators are linked to movement ofthe master manipulators or controls 100 (FIG. 4). Since the constructionand operation of such robotic manipulators are well known, their detailsneed not be repeated here. For example, general details on roboticmanipulators of this type can be found in John J. Craig, Introduction toRobotics Mechanics and Control, 2nd edition, Addison-Wesley PublishingCompany, Inc., 1989.

The kinematic component 92 is generally any device that estimates aposition, herein a “kinematic position,” of an end effector utilizinginformation available through the telesurgical system 20. In anembodiment, the kinematic component 92 utilizes kinematic positioninformation from joint states of a linkage to the end effector. As anexample, the kinematic component 92 may utilize the master/slavearchitecture for the telesurgical system 20 to calculate intendedCartesian positions of the end effectors 66, 68, 70 based upon encodersignals (from encoders 102, FIG. 4) for the joints in the linkage foreach of the tools 60, 62, 64. An example of a kinematic system isdescribed in U.S. Pat. No. 7,155,315, although others may be utilized.

As is known, during movement of a linkage, a master controller providesinstructions, for example via the vision cart 80, to the slavemanipulators to cause corresponding movement of the robot. Theinstructions provide a position (herein “command position”) in which themaster manipulators direct the slave manipulators. This command positionis the position at which the tool or end effector is ideally located asinstructed by the master manipulator, and it would be the actualposition if there were no errors in the joints and no flexing in thecomponents of the linkage.

In reality, however, an end effector or tool may be in a differentlocation than the command position, either initially or after a sequenceof moves by the slave manipulator. In some telesurgical systems,detected joint positions, provided by the encoders 102, may provideinformation about an estimated position of end effector (herein an“encoder-detected position” of the end effector). The difference betweenthe command position and the encoder-detected position may be used togenerate a joint position or kinematic error, and torque commands may beprovided for correcting the position of the end effector from theencoder-detected position to the command position, in the form of acorrection.

If a joint correction is made for joint position error as describedabove, then the new corrected position of the end effector is referredto herein as the “corrected position.” This corrected position is notnecessarily aligned with the command position, because there may beerrors in the stiffness of the joints or the readings of the encoders.In addition, even if the joints were aligned perfectly, there may besome flexion in the components of the linkage, causing the actualposition of end effector to not completely align with the controlposition.

FIG. 5 is a flowchart representing steps for calculating force on an endeffector in accordance with an embodiment. In accordance with thisembodiment, a comparison is made between a kinematic position of the endeffector versus an actual position of the end effector, and suchcomparison represents force on the end effector. In such an embodiment,the kinematic position may be the corrected position, if used, or theencoder-detected position. Since forces applied to the tool, such as astatic force experienced when the tool is pressing against anobstruction, can cause the parts of a linkage or tool to flex without adetectible change in joint states, the actual position of the endeffector may not match the kinematic position of the end effector, evenif the kinematic position is the corrected position and the correctedposition is accurate with respect to the command position. In accordancewith the embodiment shown in FIG. 5, this difference in position may beused to indicate force.

At step 500, the end effector begins at position A. Although describedas “positions” herein, a change in position may be a change in time inwhich there is no movement of the end effector. However, for ease ofdescription, “position” is used herein to mean a change of time and/orposition. At step 502, the actual position of the end effector isstored. This actual position is obtained by, for example, the tooltracking component 90. At step 504, the kinematic information for theend effector is stored. This information may be obtained, for example,via the kinematic component 92.

In accordance with an embodiment, an offset may be stored at step 506.This offset provides information regarding the difference between thekinematic information stored in step 504 and the actual positioninformation stored in step 502. Utilizing the offset, the kinematicinformation and the actual position information may be registered to thesame position.

At step 508, the end effector moves to position B. In step 510, thechange in actual position of the end effector is calculated between theactual position of the tool at position B versus the actual position ofthe tool in position A. At step 512, the change in position iscalculated using kinematic information obtained via the kinematiccomponent 92. If desired, although not required, another offset may bedetermined at position B. At step 514, the force on the tool isrepresented by the difference between the change in actual positionsbetween A and B and the change in kinematic positions between A and B.The difference between the change in actual position and the change inkinematic position is utilized to represent direction and amount offorce applied to the end effector, for example, supplied by contact ofthe end effector with body parts.

The amount of force deflection is a function of the flexibility of thetool and/or linkage, and the distance from where a force is applied tothe end effector to the exit of the cannula (body port entry). Thisinformation may be utilized to generate real force metrics using knownformulas. However, a user may also be interested in a change in force,and relative differences may be informative as to the amount of forcebeing applied.

As an example, FIG. 6 is a diagrammatic representation of movement of anend effector from position A to position B with force F resisting themovement. At position A, an image of an end effector 110 has an actualposition shown by the solid outer line for the end effector. Kinematicinformation (in this example, the corrected position) for the endeffector is represented by the dotted line 112. In the diagram shown inthe drawing, the kinematic position information matches the actualposition information. In reality, however, as described above, thekinematic position information may vary to some degree, and may notmatch unless the offset provided in step 506 is utilized. For thisexample, it is assumed that the offset is used or that the kinematicinformation matches the actual information exactly at position A. Thus,the dotted line 112, representing the kinematic position informationprovided by the kinematic component 92, matches the position of theimage 110 of the end effector, representing actual position informationprovided by the tool tracking component 90. In addition, in anembodiment, the actual position may be represented by a video of thetool.

At position B, the actual position of the end effector, represented bythe image 113, is shown as being moved from position A. This actualposition, as described above, is calculated by the tool trackingcomponent 90 (e.g., at a frame rate less than or equal to approximatelythirty frames per second). The kinematic position information, estimates(e.g., at an update cycle time of approximately 1333 Hz), however, thatthe tool, in movement from position A to position B, is now at thedotted line 114 shown with position B. The dotted line 114 represents aposition where the end effector would be if moved without force beingapplied to the end effector 110. Absent force being applied to the endeffector, this estimate is typically accurate. Although, as describedabove, kinematic position information is typically not accurate fordetermining a position of an end effector in space at a start of aprocess, the kinematic position information typically is accurate indetermining a change in position of an end effector if there is noforce.

The position shown by the dotted line 114 assumes that the beginningpoint of movement for the end effector, with respect to the kinematiccomponent 92, is the line 112. If the kinematic position information didnot match the actual position information at position A, then the offsetprovided in step 506 may be utilized at position B to project therelative position of the dotted line 114 assuming a start at line 112.

The dotted line 114 is in a different location than the actual positionof the end effector due to the difference between the kinematic positioninformation and the actual position information. The difference betweenthe two is due to force applied to the end effector in the movement fromposition A to position B. For example, in the example shown in FIG. 6, aforce F is applied to the end effector during movement. This forceprevents the end effector from moving fully as estimated by thekinematic component 92, shown by the dotted line 114. Instead, thecombination of the movement of the linkage for the end effector 110 andthe force F results in the end effector being positioned as shown by theimage 113 in FIG. 6B.

The force output 96 provided by the change in kinematic positioninformation versus actual position information may be useful for avariety of different applications. For example, the force output 96 maybe forwarded to the vision cart 80, which in turn may generateinstructions for the surgeon console 30 to create tactile feedback tothe surgeon S so that the surgeon is provided positive feedback of theexistence of force. In addition, in accordance with an embodiment and asis described above with reference to FIG. 6, the force output 96 may beutilized to generate an image representing force applied to the endeffector. For example, by displaying the diagram at the B portion ofFIG. 6, a representation of force applied on the end effector isprovided. That is, providing the visual image of where the end effectorwould be absent force (i.e., the dotted line 114), and simultaneouslydisplaying the image 113 of the actual location of the end effector, aviewer is provided a visual representation of the force applied to theend effector and the force's effect on the end effector.

In an embodiment, the force output 96 may be combined with otherinformation, such as the length the tool is inserted into the body andtool properties (e.g., via finite element analysis) to calculate theactual force that is applied on the tool. As can be understood, theamount of deflection of a tool is directly related to how much of thetool is available for flexion. The insertion length of the tool beyondcannula (body wall entry port) to the tip of the tool is readilyavailable from a robotic system. This length, together with a measureddeflection can be used to derive another quantity which is invariant toinsertion length. This quantity is one step closer to the real force,and therefore can be more intuitive to use. The other factors (such asinstrument properties) do not typically change at different instances sosurgeons can adapt to them. This new quantity can be displayed bymodulating the amount of deflection (for example, if the insertionlength is small, increase the amount of actual deflection). The exactmodulation can follow some finite element analysis. In general, however,the force output 96 is useful in that it provides relative force thatmay be useful as feedback to a surgeon S.

In an embodiment, the timing of the position A may be selected by thecomputer 82. As an example, the position A may be initiated by an event,such as closing of grippers or scissors. Alternatively, the position Amay be selected by a user, such as the surgeon S. For example, theprocess above may be initiated by a surgeon, for example by the surgeontouching a foot pedal or double-clicking the master grips of the mastercontroller when at position A so as to start the process. The surgeon'sinitiation sets position A. After force measurement or reaching positionB, a normal mode can be returned by another touch or double-click. In analternative embodiment, the process may be automated so that it occursregularly. If desired, the position A may be some combination of anevent, information that is available to the image capturing device 50,taken at regular intervals, or any combination of these. The amount oftime elapsed before establishing position B may also be determined bytime, information available, or may be requested by the surgeon S.

As an example, a surgeon may grasp an organ or other part of thepatient's body with a grasper. Position A may be initiated by thesurgeon just prior to or as grasping the organ. The grasper may thenregister the position B reading, or the surgeon may pull against theorgan, and position B may be registered after some pulling. Position Bmay be selected after a particular amount of time, or it may be selectedby the surgeon as desired. In addition, if desired, force outputprovided by the embodiments described herein may be output as a resultof a particular force being applied to the organ. This force output mayinitiate a warning or other indicator to the surgeon, for example.

As described above, the force information derived from the method ofFIG. 5 often is directed to the flexion of a tool or the linkage. Anadvantage of the method is that the force information is typically notimpacted by the body wall forces, whereas many joint sensors are. Inaddition, unlike tip force sensors, the method in FIG. 5 may senseforces along an instrument shaft (typically a Z-axis), assuming theshaft is not parallel to the viewing angle.

The display provided herein, for example, as shown in FIG. 6B, may beuseful in displaying visual information about force, regardless of theforce input. That is, the display may be used to display force sensed orotherwise provided from sources other than the computer 82.Alternatively, the force information described above may be combinedwith additional force sensing or other force information to provide moreaccurate information about force.

As an example of a different source of force information, active forcesensors may be utilized to determine the force on an end effector. Thisforce may be displayed on the display 84 without the need for kinematicinformation. Such sensors may be, for example, located at the tip of atool (i.e., at the end effector). The force information, as anotherexample, may be derived from strain gauge measurements on linkages inthe slave manipulator manipulating the tool that is being monitored, orit may be derived from encoders associated with joints in the slavemanipulator manipulating the tool that is being monitored. Such systemsfor providing force information are disclosed, for example, in U.S. Pat.App. Pub. No. US 2008/0065111 A1 (filed Sep. 29, 2007), entitled “ForceSensing for Surgical Instruments.”

As another example, force may be calculated using the kinematic errorinformation described above. In one example, the change between theencoder-detected position and the corrected position may be assumed torepresent force. Since forces applied to the tool, such as a staticforce experienced when the tool is pressing against an obstruction, cancreate a joint position error, an assumption can be made that thedifference between the two positions is a result of force on the tool.Typical processing to generate the force information may includefiltering and/or gain adjustments. As another example, the force may beextracted from the torque information generated to correct joint errors.Such systems for providing force information are disclosed in U.S. Pat.App. Pub. No. US 2005/0200324 A1 (filed Mar. 30, 2005), entitled“Non-Force Reflecting Method for Providing Tool Force Information to aUser of a Telesurgical System.”

If alternative force information is used as described above, in anembodiment, the actual force on the tool may be extracted bymathematically removing other forces, such as body wall forces and thelike. This information may be calculated, for example, by using suchinformation as the length the tool is inserted into the body and toolproperties (e.g., via finite element analysis) to extract the actualforce that is applied on the tool.

FIG. 7 is a flowchart representing steps for displaying force inaccordance with an embodiment. At step 700, an end effector begins atposition A. At step 702, the position of A is stored. At step 704, theend effector is moved to position B. At step 706, the force applied tothe end effector in the movement between position A and B is determined,for example by one of the methods described above or by other methods.At step 708, an image representing the actual position of the endeffector at position B is displayed. This image may be a video view ofthe actual end effector or another suitable image, such as a syntheticrepresentation of the end effector. At step 710, an image representingthe end effector without force being applied is displayed. Thisdisplayed image may be the dotted line 114 shown in FIG. 6B or any otherappropriate image. As an example, the display in step 710 may displayforce in a particular direction. Force information may be provided on ornear the end effector, or may be positioned in a different location,such as in another window. In any event, a user may be provided a visualor other (e.g., audible) indication of force that is applied to the endeffector.

The features described herein may be provided in stereoscopic vision sothat a user may visualize force in apparent three-dimensional form. Ascan be understood, in a stereoscopic view, force that is transverse to adirection of view is more visual in such a representation, and forcethat is parallel to a direction of view may not be displayed, andfeedback for forces in these directions may be provided by othermechanisms, such as haptic or a different type of screen display.

In addition, in accordance with an embodiment, the force informationprovided above may be provided with other force information, such assensed force information, to provide a more detailed analysis of forcebeing applied to an end effector.

Synthetic Model to Show Force

In accordance with an embodiment, instead of the dotted line 114, asynthetic image of an end effector may be displayed as a representationof the actual end effector without load. To this end, modeling data 150(FIG. 3) may be provided that is associated with the patient side cart40 and/or the computer 82. The modeling data 150 may be, for example, atwo-dimensional or three-dimensional image of the end effector. In anembodiment, such an end effector is a three-dimensional model of the endeffector and thus may represent an actual solid model of the endeffector. The modeling data 150 may be, for example, CAD data or otherthree-dimensional solid model data representing an end effector, such asthe end effector 152 shown in FIG. 8. In an embodiment, thethree-dimensional model is manipulatable at each joint so that movementsof the end effector 152 may be mimicked by a synthetic model 154 (shownin phantom line in FIG. 8) of the end effector. As shown in FIG. 8, thesynthetic model 154 may be the same size as the image of the actual endeffector 152, but it may be larger or smaller.

Although shown in dashed lines in the drawings, the synthetic model 154may be represented in a number of different ways. As an example, thesynthetic model 154 may be a transparent image of the end effector 152or a wire diagram image of the end effector. The synthetic model 154 mayalternatively be an image that is not transparent, but such a model maymake viewing of the actual end effector 152 difficult.

FIG. 9 is a flowchart representing steps for displaying the syntheticmodel 154 in accordance with an embodiment. In step 900, the endeffector 152 begins at position A. In the embodiment shown in FIG. 9,the synthetic model is displayed in accordance with the actual positioninformation (i.e., is displayed at the actual position of the endeffector 152) at step 902. Thus, the synthetic model is superimposedover the image of the end effector 152, which may be a video image ofthe end effecter. For example, as shown in FIG. 8, the synthetic model154 is translucent or transparent and may be displayed over the videoimage of the actual end effector 152. As another option, the syntheticmodel 154 may start at a location other than the actual position of theend effector 152.

At step 904, the end effector moves to position B. At step 906,kinematic position information is received for the end effector 152. Anadjustment for offset is taken at step 908, and then the synthetic model154 is displayed in step 910.

In accordance with the method in FIG. 9, the synthetic model 154 maycontinue to be updated so that force information is represented by thesynthetic model 154 and its position relative to the end effector 152.In the display shown, the end effector 152 is a video image of the endeffector. As such, steps 906-910 may be updated in real time, for boththe video image and the synthetic model 154, so that the synthetic model154 and its position are updated as the end effector 152 is moved. Insuch continual real time display of the synthetic model 154, step 902may be substituted with the display of the model at the last locationinstead of the actual position. In addition, as described above, theoffset and the original position A may be determined in accordance withan event or timing or in another manner.

In accordance with an embodiment, the methods described herein may bealtered so as to control the display of forces as the surgeon desires.For example, when a movement is made between two positions, there may bea number of other forces involved other than force on the tip of thetool, such as body cavity contact or gravity, that the surgeon does notwant to affect his or her measurement. This problem may be even morepronounced when the distance over which the end effector is movedbetween positions A and B is increased.

In general, the forces that the surgeon desires to measure during amovement of an end effector between two positions are the forces at thetip of the tool, i.e., at the end effector. In an embodiment, the forcemeasurements are taken so that they filter out, to the extent possible,forces other than those applied at the end effector. One method foreliminating other forces, such as body wall forces and gravity, is byassuming that total movement between the two positions, which indicatestotal forces on the tool between the two positions, is a combination oftwo sets of forces: those applied at the tip of the tool and otherforces. In general, body forces, gravity, and other forces that areremote of the tip may be detected by the encoders 102. As describedabove, as part of the setup process and movements for a patient sidecart and the corresponding robotic manipulator arm, the arm isinstructed to move and the encoders 102 determine whether theencoder-detected position is consistent with the command position. Ifnot, then the manipulator is instructed to move to the correctedposition. As discussed above, this movement to the corrected positionmay be used to calculate force exerted during the movement. This jointcorrection will address joint error, but typically it does not addressflexing of the components of the linkage. Assuming that flexing of thetool at the tip is the primary form of force absorption at the tip,then, in accordance with an embodiment, any force not sensed through thejoint correction process may be assumed to be force applied at the tip.Thus, in accordance with an embodiment, the forces exerted at the tipare determined by subtracting the calculated joint forces from the totalforces.

FIG. 11 is a flow chart representing steps for filtering forcesprimarily to tip forces in accordance with an embodiment. Beginning atstep 1100, and end effector is at position A. At step 1102, thekinematic correction for the linkage is updated. At step 104, the actualposition of the end effector is sensed, for example using the tooltracking component 90. At step 1106, the kinematic position iscalculated for position A. At step 1108, a kinematic force calculationis made at position A using the kinematic information, as describedabove. At step 1110, the tool is moved to Position B. At step 1112, theactual position of the tool is sensed, for example using the tooltracking component 90. At step 1114, the kinematic position iscalculated for position B. At step 1116, a force calculation is made ofthe movement from A to B using a kinematic position of the end effectorverses an actual position of the end effector, for example as describedabove with reference to FIG. 5. At step 1118, the force on the endeffector is calculated by subtracting the force in step 1112 from thetotal force from step 1116.

In the force calculation of step 1118, the force on the tool tip iscalculated based upon a difference between the total force and kinematicjoint forces. As described above, the kinematic joint forces are assumedto represent the forces other than tip forces. If desired, all threeforces (total, kinematic, or tip) or any subset of these three may begenerated as the force output 96, and/or may be displayed to a user, forexample on the display 84 or the viewer 32, as indicated by the dottedline in FIG. 4. In this calculation, it is assumed that other, outsideforces are held constant between positions A and B, and the motionbetween these two positions is due to the application of force.

Methods described herein are advantageous in that they provideinstantaneous visual feedback to a surgeon of force on a tool. A surgeonutilizing the features may grow accustomed to the intuitive feedback,and may recognize tile amount of force being used by comparing thecurrent displacement to a history of procedures. For example, a surgeonperforming a sewing procedure who sees a deflection of X may recognizethat he or she is applying roughly the same force when a deflection of Xoccurs in a different procedure.

If desired, the amount of deflection may be altered for stiff tools sothat a visual representation of the deflection is exaggerated on thedisplay. In addition, if desired, force may be displayed by showingforce information in a different manner. For example, an arrowindicating a direction of force may be used, or a syntheticrepresentation may be displayed in another direction or position.

In another aspect of the invention, the tip force information may beused to derive the deformation, or elasticity, of tissue that isclinically relevant. For, e.g., tumors on or below the tissue surface,surgeons can directly sense the difference in elasticity between normaland cancerous tissues by applying pressure to the two different types oftissue and using intuitive visualization of instrument tip flexing(described above) to determine deformation characteristics of particularlocations on the tissue. For a given amount of movement by an endeffector into contact with the tissue, the amount that the end effectoractually moves when in contact with the tissue, instead the end effectorflexing, is directly related to the deformation of the tissue. Thus,force information, as determined above, may be used to determinedeformation of a tissue. Moreover, other force information, such asprovided by sensors, may be used to determine tissue deformation. Forexample, the amount a tissue pushes back against an end effector will bereflected in the force sensed by active sensors—the more force sensed,the less the tissue is deforming. In addition, the force information,such as may be extracted about the instrument tip, may be combined with,e.g., ultrasound imaging to provide elasticity imaging of tissues andorgans underneath the tissue with absolute elasticity measurements.

In an embodiment, the display of tissue may be altered to show tissuedeformation. This deformation may be calculated, for example, based uponforce input from any number of sources, including, but not limited to,the sources listed herein. Based upon the amount of force that issensed, the deformation display may be altered accordingly. For example,in an embodiment, the tissue at an impact point, for example wheregrasped or where an end effector applies pressure, may be altered incolor, for example shaded variations in color based upon the amount offorce applied to the tissue. The tissue's point of impact can bedetermined from the location of touching tool tip that is tracked bytool tracking. The surrounding tissue surface locations (left and right)for overlaying deformation color can be obtained by sparse tissuematching and/or regular dense stereo matching constrained in theselected region of interest. The color and/or intensity of the color maybe altered based upon the sensed tip force or tissue elasticity usingexisting mechanical models. This information may be helpful, forexample, in providing visual feedback to a surgeon of force applied toparticular tissue.

For example, in FIG. 12, a method is shown for displaying deformation ofa tissue using variations in color in accordance with an embodiment.Beginning at step 1200, deformation is measured for a tissue, forexample using the methods described above. At step 1202, a determinationis made whether a minimum deformation threshold is met. Thisdetermination represents a minimum amount of deformation in which todisplay an indication of deformation for the tissue. If the threshold isnot met, then the process continually loops back to 1200. If thisthreshold is met, then the process proceeds to step 1204, where adetermination is made whether a first deformation threshold has beenmet, and if so, a first color is displayed at step 1206. In theembodiment shown in the drawings, for simplicity, this first deformationrepresents a maximum deformation permitted for the tissue, and thus thecolor may indicate such a high deformation, for example using the colorred. If this first deformation is not met, then step 1204 branches tostep 1208, where a determination is made whether a second deformationthreshold has been met, the second deformation being less than the firstdeformation. If so, then a second color is displayed at 1210. Additionaldeterminations are made, and colors applied as needed, in steps 1210 to1218, where the amount of deformation needed to meet a thresholdcontinues to decline to the minimum or until the necessary color isdisplayed. If a fourth threshold is not met, then step 1216 branches tostep 1220, where a fifth color is displayed. This color represents aminimum deformation displayed by the system. The process shown in FIG.12 may continue to loop back, so that changes in deformation arereflected by different color variations.

In the embodiment shown in the drawings, the number of different colorsdisplayed is five, although any number may be used. In addition, ifdesired, a color may be altered, such as lightened or darkened, basedupon an increase or decrease in deformation.

Using the process of FIG. 12, a surgeon is provided visual feedback ofdeformation of tissue. The amount of deformation may reflect a type oftissue and/or the condition of the tissue. Although described herein asbeing displayed as a color or color variation displayed on the tissue,deformation information may be provided in another manner or in anotherlocation.

In accordance with another embodiment, a tissue may be varied in colorbased upon how far the tissue is from an impact point. That is, thetissue may be one color at the impact point and different colors as thetissue is spaced from the impact point. The variations in colorrepresent different amounts of deformation at the different locations.To determine a color for a particular region, other techniques, such asrobust image matching or stereo matching, such as is disclosed in U.S.Pat. App. No. 61/204,082(filed Dec. 31, 2008), may be incorporated todetermine deformation at particular locations, including the impactpoint and locations spaced from the impact point.

Other variations are within the spirit of the present invention. Thus,while the invention is susceptible to various modifications andalternative constrictions, a certain illustrated embodiment thereof isshown in the drawings and has been described above in detail. It shouldbe understood, however, that there is no intention to limit theinvention to the specific form or forms disclosed, but on the contrary,the intention is to cover all modifications, alternative constructions,and equivalents falling within the spirit and scope of the invention, asdefined in the appended claims.

All references, including publications, patent applications, andpatents, cited herein are hereby incorporated by reference to the sameextent as if each reference were individually and specifically indicatedto be incorporated by reference and were set forth in its entiretyherein.

The use of the terms “a” and “an” and “the” and similar referents in thecontext of describing the invention (especially in the context of thefollowing claims) are to be construed to cover both the singular and theplural, unless otherwise indicated herein or clearly contradicted bycontext. The terms “comprising,” “having,” “including,” and “containing”are to be construed as open-ended terms (i.e., meaning “including, butnot limited to,”) unless otherwise noted. The term “connected” is to beconstrued as partly or wholly contained within, attached to, or joinedtogether, even if there is something intervening. Recitation of rangesof values herein are merely intended to serve as a shorthand method ofreferring individually to each separate value falling within the range,unless otherwise indicated herein, and each separate value isincorporated into the specification as if it were individually recitedherein. All methods described herein can be performed in any suitableorder unless otherwise indicated herein or otherwise clearlycontradicted by context. The use of any and all examples, or exemplarylanguage (e.g., “such as”) provided herein, is intended merely to betterilluminate embodiments of the invention and does not pose a limitationon the scope of the invention unless otherwise claimed. No language inthe specification should be construed as indicating any non-claimedelement as essential to the practice of the invention.

Preferred embodiments of this invention are described herein, includingthe best mode known to the inventors for carrying out the invention.Variations of those preferred embodiments may become apparent to thoseof ordinary skill in the art upon reading the foregoing description. Theinventors expect skilled artisans to employ such variations asappropriate, and the inventors intend for the invention to be practicedotherwise than as specifically described herein. Accordingly, thisinvention includes all modifications and equivalents of the subjectmatter recited in the claims appended hereto as permitted by applicablelaw. Moreover, any combination of the above-described elements in allpossible variations thereof is encompassed by the invention unlessotherwise indicated herein or otherwise clearly contradicted by context.

1. A method of providing force information for a robotic surgicalsystem, the method comprising: storing first kinematic positioninformation and first actual position information for a first positionof an end effector, the first kinematic position information being basedon joint states of a linkage used to support the end effector, the firstposition selected by either a user of the robotic surgical system orbased upon an event; storing second kinematic position information andsecond actual position information for a second position, the secondkinematic position information being based on joint states of a linkageused to support the end effector; and providing force informationregarding force applied to the end effector at the second positionutilizing the first actual position information, the second actualposition information, the first kinematic position information, and thesecond kinematic position information.
 2. The method of claim 1, whereinthe force information is derived from a positional difference between achange between the first actual position information and second actualposition information and a change between the first kinematic positioninformation and the second kinematic position information, the forceinformation comprising a difference in force applied to the end effectorbetween the first and second positions.
 3. The method of claim 2,further comprising calculating an offset between the first kinematicposition information and the first actual position information, andutilizing the offset at the second position in the derivation of theforce information.
 4. The method of claim 1, wherein the second positionis selectable by a user of the robotic surgical system.
 5. The method ofclaim 1, wherein the first actual position information and the secondactual position information each comprise visual information about theposition of the end effector.
 6. The method of claim 1, furthercomprising displaying a representation of force utilizing the forceinformation.
 7. The method of claim 6, wherein displaying arepresentation of force comprises displaying (1) a first imagerepresenting the second actual position information and (2) a secondimage representing the second kinematic position information.
 8. Themethod of claim 7, wherein the second image comprises a visualrepresentation of the end effector.
 9. The method of claim 7, whereinthe first image comprises a video image of the end effector.
 10. Themethod of claim 7, wherein the second image comprises a visualrepresentation of the second kinematic information.
 11. The method ofclaim 10, wherein the displaying a representation of force comprisesdisplaying the representation of force on a viewer of a surgeon consoleof the robotic surgical system.
 12. The method of claim 10, wherein thevisual representation comprises a synthetic representation of the tool.13. The method of claim 12, wherein the synthetic representationcomprises a model of the tool.
 14. The method of claim 13, wherein themodel comprises a three-dimensional model of the tool.
 15. The method ofclaim 6, wherein the displaying a representation of force comprisesdisplaying the representation of force on a viewer of a surgeon consoleof the robotic surgical system.
 16. The method of claim 15, wherein thesecond image comprises a visual representation of the end effector. 17.The method of claim 15, wherein the displayed representation of forcecomprises a video image of the end effector.
 18. The method of claim 1,wherein the force information is filtered to focus on forces applied ator adjacent the end effector.
 19. The method of claim 18, furthercomprising: determining joint errors from positions of the joints withrespect to a commanded position of the joints; correcting the jointstoward the command position; and utilizing the correction of the jointsto filter the force information.
 20. The method of claim 19, wherein thecorrection of the joints is utilized to generate second forceinformation, and wherein filtering the force information comprisessubtracting the second force information.
 21. The method of claim 1,wherein the force information is combined with additional forceinformation.
 22. The method of claim 21, wherein the additional forceinformation comprises information derived from active force sensors. 23.The method of claim 22, wherein the active force sensors comprise forcesensors located at the end effector.
 24. The method of claim 22, whereinthe linkage comprises one or more joints, and wherein the active forcesensors comprise sensors located at the one or more joints.
 25. Themethod of claim 22, wherein the linkage comprises joints, the methodfurther comprising: determining joint errors from positions of thejoints with respect to a commanded position of the joints; correctingthe joints toward the command position; and wherein the additional forceinformation comprises information derived from the correcting of thejoints.
 26. The method of claim 1, wherein the linkage comprises joints,and wherein the kinematic position information comprises informationabout joint states of the linkage.
 27. The method of claim 26, furthercomprising: determining joint errors from positions of the joints withrespect to a commanded position of the joints; and correcting the jointstoward the command position to a corrected position; and whereininformation about joint states of the linkage comprises informationabout the joints at the corrected position.
 28. An assembly forproviding force information within a robotic system, the assemblycomprising: a user selection component for selecting one or morepositions of an end effector at which kinematic and actual positioninformation of the end effector are to be determined; means fordetermining the kinematic position information from joint states of alinkage while the linkage supports the end effector; means fordetermining the actual position information; and means for visuallyproviding force information regarding force applied to the end effector,the force information being provided in response to the kinematicposition information and the actual position information.
 29. Theassembly of claim 28, wherein the force information is combined by themeans for visually providing force information with additional forceinformation.
 30. The assembly of claim 29, wherein the additional forceinformation comprises information derived from one or more active forcesensors.
 31. The assembly of claim 30, wherein the one or more activeforce sensors comprise one or more force sensors located at the endeffector.
 32. The assembly of claim 20, wherein the linkage comprisesone or more joints, and wherein the one or more active force sensorscomprise one or more sensors located at the one or more joints.
 33. Theassembly of claim 29, wherein the linkage comprises joints, and whereinthe additional force information comprises information derived fromcorrecting of the positions of the joints.
 34. The assembly of claim 28,wherein the means for determining actual position information receives avideo feed from an image capture device.
 35. The assembly of claim 28,wherein the means for visually providing force information includes adisplay component for displaying a representation of force utilizing theforce information.
 36. The assembly of claim 35, wherein the displaycomponent is configured to display a representation of force comprising(1) a first image representing the actual position information and (2) asecond image representing the kinematic position information.
 37. Theassembly of claim 36, wherein the second image comprises a syntheticrepresentation of the tool.
 38. The assembly of claim 37, wherein thesynthetic representation comprises a model of the tool.
 39. The assemblyof claim 38, wherein the model comprises a three-dimensional model ofthe tool.
 40. The assembly of claim 28, further comprising a filter tofocus the force information on forces applied at or adjacent the endeffector.